Ising model

 ( Spin Models ) 

The Ising model (or Lenz–Ising model), named after the physicists Ernst Ising and Wilhelm Lenz, is a mathematical model of ferromagnetism in statistical mechanics. The model consists of discrete variables that represent magnetic dipole moments of atomic "spins" that can be in one of two states (+1 or −1). The spins are arranged in a graph, usually a lattice (where the local structure repeats periodically in all directions), allowing each spin to interact with its neighbors. Neighboring spins that agree have a lower energy than those that disagree; the system tends to the lowest energy but heat disturbs this tendency, thus creating the possibility of different structural phases. The two-dimensional square-lattice Ising model is one of the simplest statistical models to show a phase transition.See , Chapters VI-VII. Though it is a highly simplified model of a magnetic material, the Ising model can still provide qualitative and sometimes quantitative results applicable to real physical systems.

The Ising model was invented by the physicist , who gave it as a problem to his student Ernst Ising. The one-dimensional Ising model was solved by alone in his 1924 thesis; Ernst Ising, Contribution to the Theory of Ferromagnetism it has no phase transition. The two-dimensional square-lattice Ising model is much harder and was only given an analytic description much later, by . It is usually solved by a transfer-matrix method, although there exists a very simple approach relating the model to a non-interacting fermionic quantum field theory.

In dimensions greater than four, the phase transition of the Ising model is described by mean-field theory. The Ising model for greater dimensions was also explored with respect to various tree topologies in the late 1970s, culminating in an exact solution of the zero-field, time-independent model for closed Cayley trees of arbitrary branching ratio, and thereby, arbitrarily large dimensionality within tree branches. The solution to this model exhibited a new, unusual phase transition behavior, along with non-vanishing long-range and nearest-neighbor spin-spin correlations, deemed relevant to large neural networks as one of its possible .

The Ising problem without an external field can be equivalently formulated as a graph maximum cut (Max-Cut) problem that can be solved via combinatorial optimization.

For any two adjacent sites $i,\; j\backslash in\backslash Lambda$ there is an interaction $J\_\{ij\}$. Also a site $j\backslash in\backslash Lambda$ has an external magnetic field $h\_j$ interacting with it. The energy of a configuration $\{\backslash sigma\}$ is given by the Hamiltonian function

$$H(\backslash sigma)\; =\; -\backslash sum\_\{\backslash langle\; ij\backslash rangle\}\; J\_\{ij\}\; \backslash sigma\_i\; \backslash sigma\_j\; -\; \backslash mu\; \backslash sum\_j\; h\_j\; \backslash sigma\_j,$$

where the first sum is over pairs of adjacent spins (every pair is counted once). The notation $\backslash langle\; ij\backslash rangle$ indicates that sites $i$ and $j$ are nearest neighbors. The magnetic moment is given by $\backslash mu$. Note that the sign in the second term of the Hamiltonian above should actually be positive because the electron's magnetic moment is antiparallel to its spin, but the negative term is used conventionally.See , Chapter 16. The Ising Hamiltonian is an example of a pseudo-Boolean function; tools from the analysis of Boolean functions can be applied to describe and study it.

The configuration probability is given by the Boltzmann distribution with inverse temperature $\backslash beta\backslash geq0$:

$$P\_\backslash beta(\backslash sigma)\; =\; \backslash frac\{e^\{-\backslash beta\; H(\backslash sigma)\}\}\{Z\_\backslash beta\},$$

where $\backslash beta\; =\; 1\; /\; (k\_\backslash text\{B\}\; T)$, and the normalization constant

$$Z\_\backslash beta\; =\; \backslash sum\_\backslash sigma\; e^\{-\backslash beta\; H(\backslash sigma)\}$$

is the partition function. For a function $f$ of the spins ("observable"), one denotes by

$$\backslash langle\; f\; \backslash rangle\_\backslash beta\; =\; \backslash sum\_\backslash sigma\; f(\backslash sigma)\; P\_\backslash beta(\backslash sigma)$$

the expectation (mean) value of $f$.

The configuration probabilities $P\_\{\backslash beta\}(\backslash sigma)$ represent the probability that (in equilibrium) the system is in a state with configuration $\backslash sigma$.

The system is called ferromagnetic or antiferromagnetic if all interactions are ferromagnetic or all are antiferromagnetic. The original Ising models were ferromagnetic, and it is still often assumed that "Ising model" means a ferromagnetic Ising model.

In a ferromagnetic Ising model, spins desire to be aligned: the configurations in which adjacent spins are of the same sign have higher probability. In an antiferromagnetic model, adjacent spins tend to have opposite signs.

The sign convention of H(σ) also explains how a spin site j interacts with the external field. Namely, the spin site wants to line up with the external field. If:

$$H(\backslash sigma)\; =\; -\backslash sum\_\{\backslash langle\; i~j\backslash rangle\}\; J\_\{ij\}\; \backslash sigma\_i\; \backslash sigma\_j.$$

When the external field is zero everywhere, h = 0, the Ising model is symmetric under switching the value of the spin in all the lattice sites; a nonzero field breaks this symmetry.

Another common simplification is to assume that all of the nearest neighbors ⟨ ij⟩ have the same interaction strength. Then we can set J_ij = J for all pairs i,  j in Λ. In this case the Hamiltonian is further simplified to

$$H(\backslash sigma)\; =\; -J\; \backslash sum\_\{\backslash langle\; i~j\backslash rangle\}\; \backslash sigma\_i\; \backslash sigma\_j.$$

For the Ising model without an external field on a graph G, the Hamiltonian becomes the following sum over the graph edges E(G)

$H(\backslash sigma)\; =\; -\backslash sum\_\{ij\backslash in\; E(G)\}\; J\_\{ij\}\backslash sigma\_i\backslash sigma\_j$.

Here each vertex i of the graph is a spin site that takes a spin value $\backslash sigma\_i\; =\; \backslash pm\; 1$. A given spin configuration $\backslash sigma$ partitions the set of vertices $V(G)$ into two $\backslash sigma$-depended subsets, those with spin up $V^+$ and those with spin down $V^-$. We denote by $\backslash delta(V^+)$ the $\backslash sigma$-depended set of edges that connects the two complementary vertex subsets $V^+$ and $V^-$. The size $\backslash left|\backslash delta(V^+)\backslash right|$ of the cut $\backslash delta(V^+)$ to bipartite graph the weighted undirected graph G can be defined as

$$\backslash left|\backslash delta(V^+)\backslash right|=\backslash frac12\backslash sum\_\{ij\backslash in\; \backslash delta(V^+)\}\; W\_\{ij\},$$

where $W\_\{ij\}$ denotes a weight of the edge $ij$ and the scaling 1/2 is introduced to compensate for double counting the same weights $W\_\{ij\}=W\_\{ji\}$.

The identities

where the total sum in the first term does not depend on $\backslash sigma$, imply that minimizing $H(\backslash sigma)$ in $\backslash sigma$ is equivalent to minimizing $\backslash sum\_\{ij\backslash in\; \backslash delta(V^+)\}\; J\_\{ij\}$. Defining the edge weight $W\_\{ij\}=-J\_\{ij\}$ thus turns the Ising problem without an external field into a graph Max-Cut problem maximizing the cut size $\backslash left|\backslash delta(V^+)\backslash right|$, which is related to the Ising Hamiltonian as follows,

$$H(\backslash sigma)\; =\; \backslash sum\_\{ij\; \backslash in\; E(G)\}\; W\_\{ij\}\; -\; 4\; \backslash left|\backslash delta(V^+)\backslash right|.$$

• In a typical configuration, are most of the spins +1 or −1, or are they split equally?
• If a spin at any given position i is 1, what is the probability that the spin at position j is also 1?
• If β is changed, is there a phase transition?
• On a lattice Λ, what is the fractal dimension of the shape of a large cluster of +1 spins?

and the system is disordered. On the basis of this result, he incorrectly concluded that this model does not exhibit phase behaviour in any dimension.

$$\backslash langle\; \backslash sigma\_i\; \backslash sigma\_j\; \backslash rangle\_\backslash beta\; \backslash geq\; c(\backslash beta)\; >\; 0.$$

This was first proven by Rudolf Peierls in 1936, using what is now called a Peierls argument.

The Ising model on a two-dimensional square lattice with no magnetic field was analytically solved by . Onsager obtained the correlation functions and free energy of the Ising model and announced the formula for the spontaneous magnetization for the 2-dimensional model in 1949 but did not give a derivation. gave the first published proof of this formula, using a limit formula for Fredholm determinants, proved in 1951 by Szegő in direct response to Onsager's work.

$$\backslash langle\; \backslash sigma\_A\; \backslash sigma\_B\; \backslash rangle\; \backslash geq\; \backslash langle\; \backslash sigma\_A\; \backslash rangle\; \backslash langle\; \backslash sigma\_B\; \backslash rangle,$$

where $\backslash langle\; \backslash sigma\_A\; \backslash rangle\; =\; \backslash langle\; \backslash prod\_\{j\; \backslash in\; A\}\; \backslash sigma\_j\; \backslash rangle$.

With $B\; =\; \backslash empty$, the special case $\backslash langle\; \backslash sigma\_A\; \backslash rangle\; \backslash ge\; 0$ results.

This means that spins are positively correlated on the Ising ferromagnet. An immediate application of this is that the magnetization of any set of spins $\backslash langle\; \backslash sigma\_A\; \backslash rangle$ is increasing with respect to any set of coupling constants $J\_B$.

$$\backslash langle\; \backslash sigma\_x\; \backslash sigma\_y\; \backslash rangle\; \backslash leq\; \backslash sum\_\{z\backslash in\; S\}\; \backslash langle\; \backslash sigma\_x\; \backslash sigma\_z\; \backslash rangle\; \backslash langle\; \backslash sigma\_z\; \backslash sigma\_y\; \backslash rangle.$$

This inequality can be used to establish the sharpness of phase transition for the Ising model.

Once modern quantum mechanics was formulated, atomism was no longer in conflict with experiment, but this did not lead to a universal acceptance of statistical mechanics, which went beyond atomism. Josiah Willard Gibbs had given a complete formalism to reproduce the laws of thermodynamics from the laws of mechanics. But many faulty arguments survived from the 19th century, when statistical mechanics was considered dubious. The lapses in intuition mostly stemmed from the fact that the limit of an infinite statistical system has many zero-one laws which are absent in finite systems: an infinitesimal change in a parameter can lead to big differences in the overall, aggregate behavior.

1. The partition function is a sum of e^{−β E} over all configurations.
2. The exponential function is everywhere analytic as a function of β.
3. The sum of analytic functions is an analytic function.

This argument works for a finite sum of exponentials, and correctly establishes that there are no singularities in the free energy of a system of a finite size. For systems which are in the thermodynamic limit (that is, for infinite systems) the infinite sum can lead to singularities. The convergence to the thermodynamic limit is fast, so that the phase behavior is apparent already on a relatively small lattice, even though the singularities are smoothed out by the system's finite size.

This was first established by Rudolf Peierls in the Ising model.

To do this, he compared the high-temperature and low-temperature limits. At infinite temperature (β = 0) all configurations have equal probability. Each spin is completely independent of any other, and if typical configurations at infinite temperature are plotted so that plus/minus are represented by black and white, they look like television snow. For high, but not infinite temperature, there are small correlations between neighboring positions, the snow tends to clump a little bit, but the screen stays randomly looking, and there is no net excess of black or white.

A quantitative measure of the excess is the magnetization, which is the average value of the spin:

$$M\; =\; \backslash frac\{1\}\{N\}\; \backslash sum\_\{i=1\}^N\; \backslash sigma\_i.$$

A bogus argument analogous to the argument in the last section now establishes that the average magnetization in the Ising model is always zero.

1. Every configuration of spins has equal energy to the configuration with all spins flipped.
2. So for every configuration with magnetization M there is a configuration with magnetization − M with equal probability.
3. The system should therefore spend equal amounts of time in the configuration with magnetization M as with magnetization − M.
4. So the average magnetization (over all time) is zero.

As before, this only proves that the average magnetization is zero at any finite volume. For an infinite system, fluctuations might not be able to push the system from a mostly plus state to a mostly minus with a nonzero probability.

For very high temperatures, the magnetization is zero, as it is at infinite temperature. To see this, note that if spin A has only a small correlation ε with spin B, and B is only weakly correlated with C, but C is otherwise independent of A, the amount of correlation of A and C goes like ε². For two spins separated by distance L, the amount of correlation goes as ε ^L, but if there is more than one path by which the correlations can travel, this amount is enhanced by the number of paths.

The number of paths of length L on a square lattice in d dimensions is $$N(L)\; =\; (2d)^L,$$ since there are 2 d choices for where to go at each step.

A bound on the total correlation is given by the contribution to the correlation by summing over all paths linking two points, which is bounded above by the sum over all paths of length L divided by $$\backslash sum\_L\; (2d)^L\; \backslash varepsilon^L,$$ which goes to zero when ε is small.

At low temperatures (β ≫ 1) the configurations are near the lowest-energy configuration, the one where all the spins are plus or all the spins are minus. Peierls asked whether it is statistically possible at low temperature, starting with all the spins minus, to fluctuate to a state where most of the spins are plus. For this to happen, droplets of plus spin must be able to congeal to make the plus state.

The energy of a droplet of plus spins in a minus background is proportional to the perimeter of the droplet L, where plus spins and minus spins neighbor each other. For a droplet with perimeter L, the area is somewhere between ( L − 2)/2 (the straight line) and ( L/4)² (the square box). The probability cost for introducing a droplet has the factor e^{−β L}, but this contributes to the partition function multiplied by the total number of droplets with perimeter L, which is less than the total number of paths of length L: So that the total spin contribution from droplets, even overcounting by allowing each site to have a separate droplet, is bounded above by $$\backslash sum\_L\; L^2\; 4^\{2L\}\; e^\{-4\backslash beta\; L\},$$

which goes to zero at large β. For β sufficiently large, this exponentially suppresses long loops, so that they cannot occur, and the magnetization never fluctuates too far from −1.

So Peierls established that the magnetization in the Ising model eventually defines superselection sectors, separated domains not linked by finite fluctuations.

In the 19th century, it was thought that magnetic fields are due to currents in matter, and Ampère postulated that permanent magnets are caused by permanent atomic currents. The motion of classical charged particles could not explain permanent currents though, as shown by Joseph Larmor. In order to have ferromagnetism, the atoms must have permanent which are not due to the motion of classical charges.

Once the electron's spin was discovered, it was clear that the magnetism should be due to a large number of electron spins all oriented in the same direction. It was natural to ask how the electrons' spins all know which direction to point in, because the electrons on one side of a magnet don't directly interact with the electrons on the other side. They can only influence their neighbors. The Ising model was designed to investigate whether a large fraction of the electron spins could be oriented in the same direction using only local forces.

A coarse model is to make space-time a lattice and imagine that each position either contains an atom or it doesn't. The space of configuration is that of independent bits B_i, where each bit is either 0 or 1 depending on whether the position is occupied or not. An attractive interaction reduces the energy of two nearby atoms. If the attraction is only between nearest neighbors, the energy is reduced by −4 JB _i B _j for each occupied neighboring pair.

The density of the atoms can be controlled by adding a chemical potential, which is a multiplicative probability cost for adding one more atom. A multiplicative factor in probability can be reinterpreted as an additive term in the logarithm – the energy. The extra energy of a configuration with N atoms is changed by μN. The probability cost of one more atom is a factor of exp(− βμ).

So the energy of the lattice gas is: $$E\; =\; -\; \backslash frac\{1\}\{2\}\; \backslash sum\_\{\backslash langle\; i,j\; \backslash rangle\}\; 4\; J\; B\_i\; B\_j\; +\; \backslash sum\_i\; \backslash mu\; B\_i.$$

Rewriting the bits in terms of spins, $B\_i\; =\; (S\_i\; +\; 1)/2.$ $$E\; =\; -\; \backslash frac\{1\}\{2\}\; \backslash sum\_\{\backslash langle\; i,j\; \backslash rangle\}\; J\; S\_i\; S\_j\; -\; \backslash frac\{1\}\{2\}\; \backslash sum\_i\; (4\; J\; -\; \backslash mu)\; S\_i.$$

For lattices where every site has an equal number of neighbors, this is the Ising model with a magnetic field h = ( zJ −  μ)/2, where z is the number of neighbors.

In biological systems, modified versions of the lattice gas model have been used to understand a range of binding behaviors. These include the binding of ligands to receptors in the cell surface, the binding of chemotaxis proteins to the flagellar motor, and the condensation of DNA.

Following the general approach of Jaynes, a later interpretation of Schneidman, Berry, Segev and Bialek, is that the Ising model is useful for any model of neural function, because a statistical model for neural activity should be chosen using the principle of maximum entropy. Given a collection of neurons, a statistical model which can reproduce the average firing rate for each neuron introduces a Lagrange multiplier for each neuron: $$E\; =\; -\; \backslash sum\_i\; h\_i\; S\_i$$ But the activity of each neuron in this model is statistically independent. To allow for pair correlations, when one neuron tends to fire (or not to fire) along with another, introduce pair-wise lagrange multipliers: $$E=\; -\; \backslash tfrac\{1\}\{2\}\; \backslash sum\_\{ij\}\; J\_\{ij\}\; S\_i\; S\_j\; -\; \backslash sum\_i\; h\_i\; S\_i$$ where $J\_\{ij\}$ are not restricted to neighbors. Note that this generalization of Ising model is sometimes called the quadratic exponential binary distribution in statistics. This energy function only introduces probability biases for a spin having a value and for a pair of spins having the same value. Higher order correlations are unconstrained by the multipliers. An activity pattern sampled from this distribution requires the largest number of bits to store in a computer, in the most efficient coding scheme imaginable, as compared with any other distribution with the same average activity and pairwise correlations. This means that Ising models are relevant to any system which is described by bits which are as random as possible, with constraints on the pairwise correlations and the average number of 1s, which frequently occurs in both the physical and social sciences.

The Sherrington–Kirkpatrick model of spin glass, published in 1975, is the Hopfield network with random initialization. Sherrington and Kirkpatrick found that it is highly likely for the energy function of the SK model to have many local minima. In the 1982 paper, Hopfield applied this recently developed theory to study the Hopfield network with binary activation functions. In a 1984 paper he extended this to continuous activation functions. It became a standard model for the study of neural networks through statistical mechanics.

$$-\backslash beta\; f\; =\; \backslash ln\; 2\; +\; \backslash frac\{2\backslash gamma\}\{(\backslash gamma+1)\}\; \backslash ln\; (\backslash cosh\; J)\; +\; \backslash frac\{\backslash gamma(\backslash gamma-1)\}\{(\backslash gamma+1)\}\; \backslash sum\_\{i=2\}^z\backslash frac\{1\}\{\backslash gamma^i\}\backslash ln\; J\_i\; (\backslash tau)$$

where $\backslash gamma$ is an arbitrary branching ratio (greater than or equal to 2), $t\; \backslash equiv\; \backslash tanh\; J$, $\backslash tau\; \backslash equiv\; t^2$, $J\; \backslash equiv\; \backslash beta\backslash epsilon$ (with $\backslash epsilon$ representing the nearest-neighbor interaction energy) and there are k (→ ∞ in the thermodynamic limit) generations in each of the tree branches (forming the closed tree architecture as shown in the given closed Cayley tree diagram.) The sum in the last term can be shown to converge uniformly and rapidly (i.e. for z → ∞, it remains finite) yielding a continuous and monotonous function, establishing that, for $\backslash gamma$ greater than or equal to 2, the free energy is a continuous function of temperature T. Further analysis of the free energy indicates that it exhibits an unusual discontinuous first derivative at the critical temperature (, .)

The spin-spin correlation between sites (in general, m and n) on the tree was found to have a transition point when considered at the vertices (e.g. A and Ā, its reflection), their respective neighboring sites (such as B and its reflection), and between sites adjacent to the top and bottom extreme vertices of the two trees (e.g. A and B), as may be determined from $$\backslash langle\; s\_m\; s\_n\; \backslash rangle\; =\; \{Z\_N\}^\{-1\}(0,T)\; \backslash cosh^\{N\_b\}\; 2^N\; \backslash sum\_\{l=1\}^z\; g\_\{mn\}(l)\; t^l$$ where $N\_b$ is equal to the number of bonds, $g\_\{mn\}(l)t^l$ is the number of graphs counted for odd vertices with even intermediate sites (see cited methodologies and references for detailed calculations), $2^N$ is the multiplicity resulting from two-valued spin possibilities and the partition function $\{Z\_N\}$ is derived from $\backslash sum\_\{\backslash \{s\backslash \}\}e^\{-\backslash beta\; H\}$. (Note: $s\_i$ is consistent with the referenced literature in this section and is equivalent to $S\_i$ or $\backslash sigma\_i$ utilized above and in earlier sections; it is valued at $\backslash pm\; 1$.) The critical temperature $T\_C$ is given by $$T\_C\; =\; \backslash frac\{2\backslash epsilon\}\{k\_\backslash text\{B\}\backslash ln(\backslash sqrt\}.$$

The critical temperature for this model is only determined by the branching ratio $\backslash gamma$ and the site-to-site interaction energy $\backslash epsilon$, a fact which may have direct implications associated with neural structure vs. its function (in that it relates the energies of interaction and branching ratio to its transitional behavior.) For example, a relationship between the transition behavior of activities of neural networks between sleeping and wakeful states (which may correlate with a spin-spin type of phase transition) in terms of changes in neural interconnectivity ($\backslash gamma$) and/or neighbor-to-neighbor interactions ($\backslash epsilon$), over time, is just one possible avenue suggested for further experimental investigation into such a phenomenon. In any case, for this Ising model it was established, that “the stability of the long-range correlation increases with increasing $\backslash gamma$ or increasing $\backslash epsilon$.”

For this topology, the spin-spin correlation was found to be zero between the extreme vertices and the central sites at which the two trees (or branches) are joined (i.e. between A and individually C, D, or E.) This behavior is explained to be due to the fact that, as k increases, the number of links increases exponentially (between the extreme vertices) and so even though the contribution to spin correlations decrease exponentially, the correlation between sites such as the extreme vertex (A) in one tree and the extreme vertex in the joined tree (Ā) remains finite (above the critical temperature.) In addition, A and B also exhibit a non-vanishing correlation (as do their reflections) thus lending itself to, for B level sites (with A level), being considered “clusters” which tend to exhibit synchronization of firing.

Based upon a review of other classical network models as a comparison, the Ising model on a closed Cayley tree was determined to be the first classical statistical mechanical model to demonstrate both local and long-range sites with non-vanishing spin-spin correlations, while at the same time exhibiting intermediate sites with zero correlation, which indeed was a relevant matter for large neural networks at the time of its consideration. The model's behavior is also of relevance for any other divergent-convergent tree physical (or biological) system exhibiting a closed Cayley tree topology with an Ising-type of interaction. This topology should not be ignored since its behavior for Ising models has been solved exactly, and presumably nature will have found a way of taking advantage of such simple symmetries at many levels of its designs.

early on noted the possibility of interrelationships between (1) the classical large neural network model (with similar coupled divergent-convergent topologies) with (2) an underlying statistical quantum mechanical model (independent of topology and with persistence in fundamental quantum states):
     

It was a natural and common belief among early neurophysicists (e.g. Umezawa, Krizan, Barth, etc.) that classical neural models (including those with statistical mechanical aspects) will one day have to be integrated with quantum physics (with quantum statistical aspects), similar perhaps to how the domain of chemistry has historically integrated itself into quantum physics via quantum chemistry.

Several additional statistical mechanical problems of interest remain to be solved for the closed Cayley tree, including the time-dependent case and the external field situation, as well as theoretical efforts aimed at understanding interrelationships with underlying quantum constituents and their physics.

L = |Λ|: the total number of sites on the lattice,

σ _j ∈ {−1, +1}: an individual spin site on the lattice, j = 1, ..., L,

S ∈ {−1, +1} ^L: state of the system.

Since every spin site has ±1 spin, there are 2 ^L different states that are possible.

e^{-\beta(H_\nu - H_\mu)}, & \text{if } H_\nu - H_\mu > 0, \\
1 & \text{otherwise}.
     

e^{\beta(h+J)} & e^{-\beta J} \\
e^{-\beta J} & e^{-\beta(h-J)}
     

obtained the following analytical expression for the free energy of the Ising model on the anisotropic square lattice when the magnetic field $h=0$ in the thermodynamic limit as a function of temperature and the horizontal and vertical interaction energies $J\_1$ and $J\_2$, respectively